Reports as jobs and as automations - #2297
Draft
Flix6x wants to merge 16 commits into
Draft
Conversation
- Activate the reporting queue (it was prepared but commented out), including worker help texts and queue cleanup. - Reporters accept as_job: a job is queued (with trigger meta data) that rebuilds the reporter from its data source, computes the report and saves the results to the database. - `flexmeasures add report --as-job` queues such a job; reporting jobs show up in the asset's jobs overview (status page and API). Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Automations can now compute reports on a recurring basis: - `flexmeasures add automation --type reports --reporter <class>` stores the reporter config on a data source (steady across runs, so all report results attribute to the same source) and validates the report parameters. - The report window resolves freshly on each run: 'start-offset'/'end-offset' fields (comma-separated Pandas offsets, applied to the run time in the first output sensor's timezone) express a rolling window, and without any timing fields the window defaults to the last cron period (from the previous cron fire time until the run time). Absolute start/end still work, but draw a warning. - The API creation field 'forecaster' is generalized to 'generator' (also accepting reporter classes), and the UI's New automation modal gains data generator and config fields; the Reports tab is now enabled. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Documentation build overview
141 files changed ·
|
Each automation run is recorded in Redis; a report automation without timing fields then reports on the period since its actual last run, falling back to the last cron period when no last run is known (e.g. on the first run, or after a Redis flush). This gives gapless coverage even when runs are missed. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Consolidates the shared automations concept (model, lifecycle, runner deployment, provenance) into documentation/features/automations.rst, with the per-feature pages linking to it and keeping only their type-specific parameter semantics. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
This was referenced Jul 11, 2026
- API automation creation now checks that the caller may read every sensor referenced in the parameters/config and record data on the sensors the automation writes to, closing a cross-account data read/write hole. - `add report --as-job` implies --save-config (the worker rebuilds the reporter from its data source, so jobs without stored config always crashed). - Report automation parameters are validated with the chosen reporter's own parameters schema, not the base schema. - Invalid start-offset/end-offset strings are rejected at creation instead of being silently skipped at run time (which yielded empty report windows). - run_report_job wipes the shared cached reporter's parameter state, like the automation runner already did, so consecutive jobs in one worker process don't pollute each other. - Default report windows now anchor to the end of the last *successfully* covered window, recorded by the reporting job upon success — failed jobs no longer create permanent reporting gaps, and the enqueue-time minute-rollover gap is gone (the recorded anchor is the window end itself). - The cron-period fallback window is computed in the platform timezone, matching how the runner decides when automations fire. - Job stats for schedules automations also scan flex-model device sensors (which may belong to child assets), so failed per-device jobs show up. - The trigger provenance kwarg is excluded from the job cache hash, so identical schedule requests from different origins dedupe again. Part of #2288 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX
Merge the current automation CRUD foundation into the report automation branch so report jobs share the established timezone, catch-up, permission, API, and UI behavior. The conflict resolution keeps report-specific queues, CLI options, views, documentation, and tests while retaining the newer scheduling and sensor-access invariants from the lower stack. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Make reporters declare the sensors they read from and write to so automation creation, detail rendering, and sensor links use one authoritative dependency model. Profit and loss reports now include price sensors stored in reporter configuration, preventing cross-organisation report automations from bypassing access checks, and the incomplete duplicate scanner in the API is removed. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Require every report payload to declare at least one output and preserve the inherited required constraints when specialized profit and aggregation schemas narrow list lengths. Invalid automations now fail during API or CLI validation instead of being accepted and later crashing a reporting worker with missing method arguments. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Apply the existing automation subtree invariant to report outputs during both creation and execution, after access checks have established that sensor metadata may be discussed. Reporter parameters are prepared before dependency resolution so recurring reports with relative windows expose and validate their sensors consistently on API details and worker runs. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Extend the database invariant that protects executable automations so report rows, like forecast rows, cannot exist without a data generator. The migration replaces the forecast-only check constraint while preserving generator-free schedule automations and provides a reversible downgrade to the previous rule. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Build recurring report windows from the canonical cron occurrence claimed by the runner and interpret prior occurrences in each automation's own IANA timezone. Delayed catch-up runs now produce stable boundaries, including across daylight-saving gaps, instead of using the platform timezone and the worker's later wall-clock time. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Update the Redis coverage anchor with an optimistic transaction that only accepts a later report end. Concurrent or out-of-order reporting workers can no longer let an older completion rewind the next default window and cause already reported periods to be processed again. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Assert that the asset automations page exposes the reports tab and its dedicated table alongside forecasts and schedules. This protects the report UI added by the stacked branch from disappearing during future template reconciliations. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Describe per-automation timezone selection and the successful-coverage model used by recurring reports. The documentation now distinguishes the claimed cron occurrence from delayed runner time and explains how first runs, daylight-saving transitions, and out-of-order worker completions determine report windows. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
Commit the fixture-owned sensor setup after rejected report automation requests so the shared API test database does not retain an idle transaction during teardown. This keeps the complete automation API module deterministic while preserving assertions that no unauthorized automation was created. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
BelhsanHmida
added a commit
that referenced
this pull request
Aug 12, 2026
Context: - PR #2300 is stacked on #2297, whose automation, timezone, permission, and CLI validation changes advanced after the template branch diverged. Change: - Merge the updated report-automation branch while preserving prepared-template commands and combining them with the current automation CLI validation and output-scope invariant. Signed-off-by: Mohamed Belhsan Hmida <mohamedbelhsanhmida@gmail.com>
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Milestone 4 of #2288 (stacked on #2294): reports can run as background jobs, and automations can compute them on a recurring basis.
Reports as jobs
run-workersupport and queue cleanup.as_job: a job is queued that rebuilds the reporter from its data source, computes the report and saves the results.flexmeasures add report --as-jobuses it; reporting jobs appear in the asset's jobs overview with their Created Via provenance.Reports as automations — rolling window design
Following the data-generator philosophy: the reporter class + config live on a steady data source (same ID, immutable row, pinned via the automation's
generator_id), while the automation'sparametersJSONB holds the per-run parameters whose timing resolves freshly on each run:start-offset/end-offsetfields (comma-separated offsets incl.DBday-begin, e.g."-1D,DB"), applied to the run time in the first output sensor's timezone — exact parity withadd report --start-offset.start/endstill validate but draw a warning (each run would compute the same period).Also
forecasteris generalized togenerator(accepts forecaster or reporter classes); the UI's New automation modal gains Data generator and config fields, and the Reports tab is enabled.How to test
Tests:
pytest flexmeasures/cli/tests/test_automations.pyincludes a full roundtrip (create → runner queues reporting job with provenance → worker computes and saves the report, values verified).🤖 Generated with Claude Code
https://claude.ai/code/session_01Rbix8k1JfeUWNXEmHEZVpX